Skip to content

Update Spock Tests to run with STRICT_TEST mode by default#11032

Open
mhlidd wants to merge 17 commits intomasterfrom
mhlidd/run_spock_with_strict
Open

Update Spock Tests to run with STRICT_TEST mode by default#11032
mhlidd wants to merge 17 commits intomasterfrom
mhlidd/run_spock_with_strict

Conversation

@mhlidd
Copy link
Copy Markdown
Contributor

@mhlidd mhlidd commented Apr 1, 2026 •
edited
Loading

What Does This Do

This PR updates Spock tests to run with STRICT_TEST by default. This means that all Environment Variables that are queried but not documented will fail the test. We fail the test by throwing an error when querying such environment variable, but if the test has automatic error handling we also throw an error during test cleanup to log the missing configurations from supported-configurations.json.

If a test is using fake configurations, the test itself can override the strictness mode with TEST to bypass this mechanism. Note that this is being done w/ many Instrumentation tests as they create fake decorators that register fake configs.

Motivation

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

mhlidd added 14 commits April 1, 2026 15:38
@mhlidd
updating config helper to have descriptive exception in strict mode
4c8153c
@mhlidd
enabling ConfigHelper strict mode by default for spock tests
f29cdf5
@mhlidd
updating test-utils to include config-utils dependency at runtime
1a30830
@mhlidd
overriding strictness for tests using fake env vars
da699c2
@mhlidd
test adding real env vars
650e7d5
@mhlidd
updating tests using fake env vars and adding more env vars
14e0405
@mhlidd
overriding test mode for elasticsearch and opensearch
b998bec
@mhlidd
adding final integration configs and overriding fake config tests
3e2ccfe
@mhlidd
updating strictness in configurePreAgent
92e6d26
@mhlidd
update config helper to guarantee throw an error that won't be caught
3582e00
@mhlidd
configuring more fake config tests
e7b23e9
@mhlidd
adding final configurations
b618154
@mhlidd
Merge branch 'master' into mhlidd/run_spock_with_strict
84dc914
@mhlidd
revert supported-configurations
9d52790
@mhlidd mhlidd changed the base branch from master to mhlidd/update_strictness_policy April 2, 2026 21:09
@mhlidd mhlidd changed the base branch from mhlidd/update_strictness_policy to master April 2, 2026 21:10
@mhlidd mhlidd mentioned this pull request Apr 5, 2026
Merged
@pr-commenter
Copy link
Copy Markdown

pr-commenter bot commented Apr 5, 2026 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mhlidd/run_spock_with_strict
git_commit_date 1775497568 1775503219
git_commit_sha b664cc0 64af647
release_version 1.61.0-SNAPSHOT~b664cc0f2a 1.61.0-SNAPSHOT~64af64756a
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1775505076 1775505076
ci_job_id 1570881599 1570881599
ci_pipeline_id 106256212 106256212
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-0k3buref 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-0k3buref 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 12 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~64af64756a, baseline=1.61.0-SNAPSHOT~b664cc0f2a

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.054 s) : 0, 1053875
Total [baseline] (8.835 s) : 0, 8835228
Agent [candidate] (1.058 s) : 0, 1057872
Total [candidate] (8.873 s) : 0, 8873297
section iast
Agent [baseline] (1.224 s) : 0, 1223539
Total [baseline] (9.578 s) : 0, 9577752
Agent [candidate] (1.224 s) : 0, 1223749
Total [candidate] (9.553 s) : 0, 9553081
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.054 s -
Agent iast 1.224 s 169.664 ms (16.1%)
Total tracing 8.835 s -
Total iast 9.578 s 742.524 ms (8.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.058 s -
Agent iast 1.224 s 165.877 ms (15.7%)
Total tracing 8.873 s -
Total iast 9.553 s 679.785 ms (7.7%) 
gantt
    title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~64af64756a, baseline=1.61.0-SNAPSHOT~b664cc0f2a

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.235 ms) : 0, 1235
crashtracking [candidate] (1.228 ms) : 0, 1228
BytebuddyAgent [baseline] (632.124 ms) : 0, 632124
BytebuddyAgent [candidate] (633.795 ms) : 0, 633795
AgentMeter [baseline] (29.381 ms) : 0, 29381
AgentMeter [candidate] (29.287 ms) : 0, 29287
GlobalTracer [baseline] (248.854 ms) : 0, 248854
GlobalTracer [candidate] (248.548 ms) : 0, 248548
AppSec [baseline] (31.97 ms) : 0, 31970
AppSec [candidate] (31.987 ms) : 0, 31987
Debugger [baseline] (59.071 ms) : 0, 59071
Debugger [candidate] (59.142 ms) : 0, 59142
Remote Config [baseline] (602.367 µs) : 0, 602
Remote Config [candidate] (598.592 µs) : 0, 599
Telemetry [baseline] (7.997 ms) : 0, 7997
Telemetry [candidate] (8.1 ms) : 0, 8100
Flare Poller [baseline] (6.48 ms) : 0, 6480
Flare Poller [candidate] (8.958 ms) : 0, 8958
section iast
crashtracking [baseline] (1.224 ms) : 0, 1224
crashtracking [candidate] (1.223 ms) : 0, 1223
BytebuddyAgent [baseline] (801.09 ms) : 0, 801090
BytebuddyAgent [candidate] (801.354 ms) : 0, 801354
AgentMeter [baseline] (11.379 ms) : 0, 11379
AgentMeter [candidate] (11.369 ms) : 0, 11369
GlobalTracer [baseline] (239.315 ms) : 0, 239315
GlobalTracer [candidate] (239.159 ms) : 0, 239159
AppSec [baseline] (33.58 ms) : 0, 33580
AppSec [candidate] (30.835 ms) : 0, 30835
Debugger [baseline] (55.452 ms) : 0, 55452
Debugger [candidate] (58.423 ms) : 0, 58423
Remote Config [baseline] (519.942 µs) : 0, 520
Remote Config [candidate] (523.801 µs) : 0, 524
Telemetry [baseline] (13.994 ms) : 0, 13994
Telemetry [candidate] (14.34 ms) : 0, 14340
Flare Poller [baseline] (4.762 ms) : 0, 4762
Flare Poller [candidate] (4.403 ms) : 0, 4403
IAST [baseline] (26.079 ms) : 0, 26079
IAST [candidate] (25.788 ms) : 0, 25788
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~64af64756a, baseline=1.61.0-SNAPSHOT~b664cc0f2a

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.063 s) : 0, 1062609
Total [baseline] (11.119 s) : 0, 11119344
Agent [candidate] (1.054 s) : 0, 1054231
Total [candidate] (11.046 s) : 0, 11046307
section appsec
Agent [baseline] (1.251 s) : 0, 1251281
Total [baseline] (11.22 s) : 0, 11220023
Agent [candidate] (1.246 s) : 0, 1245501
Total [candidate] (11.159 s) : 0, 11158878
section iast
Agent [baseline] (1.242 s) : 0, 1241543
Total [baseline] (11.419 s) : 0, 11419494
Agent [candidate] (1.253 s) : 0, 1253106
Total [candidate] (11.361 s) : 0, 11361354
section profiling
Agent [baseline] (1.185 s) : 0, 1185153
Total [baseline] (11.135 s) : 0, 11135239
Agent [candidate] (1.192 s) : 0, 1191604
Total [candidate] (11.095 s) : 0, 11094726
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.063 s -
Agent appsec 1.251 s 188.672 ms (17.8%)
Agent iast 1.242 s 178.934 ms (16.8%)
Agent profiling 1.185 s 122.544 ms (11.5%)
Total tracing 11.119 s -
Total appsec 11.22 s 100.68 ms (0.9%)
Total iast 11.419 s 300.15 ms (2.7%)
Total profiling 11.135 s 15.895 ms (0.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.054 s -
Agent appsec 1.246 s 191.27 ms (18.1%)
Agent iast 1.253 s 198.875 ms (18.9%)
Agent profiling 1.192 s 137.373 ms (13.0%)
Total tracing 11.046 s -
Total appsec 11.159 s 112.572 ms (1.0%)
Total iast 11.361 s 315.047 ms (2.9%)
Total profiling 11.095 s 48.419 ms (0.4%) 
gantt
    title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~64af64756a, baseline=1.61.0-SNAPSHOT~b664cc0f2a

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.237 ms) : 0, 1237
crashtracking [candidate] (1.215 ms) : 0, 1215
BytebuddyAgent [baseline] (637.346 ms) : 0, 637346
BytebuddyAgent [candidate] (631.466 ms) : 0, 631466
AgentMeter [baseline] (29.521 ms) : 0, 29521
AgentMeter [candidate] (29.384 ms) : 0, 29384
GlobalTracer [baseline] (250.169 ms) : 0, 250169
GlobalTracer [candidate] (248.776 ms) : 0, 248776
AppSec [baseline] (31.97 ms) : 0, 31970
AppSec [candidate] (32.087 ms) : 0, 32087
Debugger [baseline] (60.052 ms) : 0, 60052
Debugger [candidate] (59.942 ms) : 0, 59942
Remote Config [baseline] (596.386 µs) : 0, 596
Remote Config [candidate] (624.784 µs) : 0, 625
Telemetry [baseline] (8.068 ms) : 0, 8068
Telemetry [candidate] (8.047 ms) : 0, 8047
Flare Poller [baseline] (7.297 ms) : 0, 7297
Flare Poller [candidate] (6.63 ms) : 0, 6630
section appsec
crashtracking [baseline] (1.236 ms) : 0, 1236
crashtracking [candidate] (1.209 ms) : 0, 1209
BytebuddyAgent [baseline] (661.375 ms) : 0, 661375
BytebuddyAgent [candidate] (660.756 ms) : 0, 660756
AgentMeter [baseline] (12.115 ms) : 0, 12115
AgentMeter [candidate] (11.998 ms) : 0, 11998
GlobalTracer [baseline] (250.228 ms) : 0, 250228
GlobalTracer [candidate] (248.516 ms) : 0, 248516
IAST [baseline] (24.79 ms) : 0, 24790
IAST [candidate] (24.553 ms) : 0, 24553
AppSec [baseline] (185.528 ms) : 0, 185528
AppSec [candidate] (183.416 ms) : 0, 183416
Debugger [baseline] (66.636 ms) : 0, 66636
Debugger [candidate] (66.103 ms) : 0, 66103
Remote Config [baseline] (610.685 µs) : 0, 611
Remote Config [candidate] (612.965 µs) : 0, 613
Telemetry [baseline] (8.727 ms) : 0, 8727
Telemetry [candidate] (8.557 ms) : 0, 8557
Flare Poller [baseline] (3.6 ms) : 0, 3600
Flare Poller [candidate] (3.506 ms) : 0, 3506
section iast
crashtracking [baseline] (1.244 ms) : 0, 1244
crashtracking [candidate] (1.26 ms) : 0, 1260
BytebuddyAgent [baseline] (812.764 ms) : 0, 812764
BytebuddyAgent [candidate] (825.217 ms) : 0, 825217
AgentMeter [baseline] (11.598 ms) : 0, 11598
AgentMeter [candidate] (12.18 ms) : 0, 12180
GlobalTracer [baseline] (242.213 ms) : 0, 242213
GlobalTracer [candidate] (241.915 ms) : 0, 241915
IAST [baseline] (26.246 ms) : 0, 26246
IAST [candidate] (26.147 ms) : 0, 26147
AppSec [baseline] (30.823 ms) : 0, 30823
AppSec [candidate] (33.531 ms) : 0, 33531
Debugger [baseline] (60.804 ms) : 0, 60804
Debugger [candidate] (56.724 ms) : 0, 56724
Remote Config [baseline] (524.471 µs) : 0, 524
Remote Config [candidate] (537.91 µs) : 0, 538
Telemetry [baseline] (14.306 ms) : 0, 14306
Telemetry [candidate] (14.356 ms) : 0, 14356
Flare Poller [baseline] (4.632 ms) : 0, 4632
Flare Poller [candidate] (4.249 ms) : 0, 4249
section profiling
crashtracking [baseline] (1.184 ms) : 0, 1184
crashtracking [candidate] (1.189 ms) : 0, 1189
BytebuddyAgent [baseline] (691.708 ms) : 0, 691708
BytebuddyAgent [candidate] (697.007 ms) : 0, 697007
AgentMeter [baseline] (9.17 ms) : 0, 9170
AgentMeter [candidate] (9.127 ms) : 0, 9127
GlobalTracer [baseline] (206.953 ms) : 0, 206953
GlobalTracer [candidate] (207.936 ms) : 0, 207936
AppSec [baseline] (32.653 ms) : 0, 32653
AppSec [candidate] (32.622 ms) : 0, 32622
Debugger [baseline] (65.999 ms) : 0, 65999
Debugger [candidate] (66.229 ms) : 0, 66229
Remote Config [baseline] (575.743 µs) : 0, 576
Remote Config [candidate] (564.679 µs) : 0, 565
Telemetry [baseline] (7.924 ms) : 0, 7924
Telemetry [candidate] (7.958 ms) : 0, 7958
Flare Poller [baseline] (3.565 ms) : 0, 3565
Flare Poller [candidate] (3.635 ms) : 0, 3635
ProfilingAgent [baseline] (94.159 ms) : 0, 94159
ProfilingAgent [candidate] (93.771 ms) : 0, 93771
Profiling [baseline] (94.726 ms) : 0, 94726
Profiling [candidate] (94.338 ms) : 0, 94338
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mhlidd/run_spock_with_strict
git_commit_date 1775497568 1775503219
git_commit_sha b664cc0 64af647
release_version 1.61.0-SNAPSHOT~b664cc0f2a 1.61.0-SNAPSHOT~64af64756a
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1775505636 1775505636
ci_job_id 1570881600 1570881600
ci_pipeline_id 106256212 106256212
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-gtpvqzto 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-gtpvqzto 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 3 performance improvements and 2 performance regressions! Performance is the same for 16 metrics, 15 unstable metrics.

scenario Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p95 Δ mean throughput candidate mean agg_http_req_duration_p50 candidate mean agg_http_req_duration_p95 candidate mean throughput baseline mean agg_http_req_duration_p50 baseline mean agg_http_req_duration_p95 baseline mean throughput
scenario:load:insecure-bank:profiling:high_load better
[-219.611µs; -52.199µs] or [-11.240%; -2.672%]		 unstable
[-1070.987µs; -204.938µs] or [-17.654%; -3.378%]		 unstable
[+0.470op/s; +434.530op/s] or [+0.026%; +24.422%]		 1.818ms 5.429ms 1996.781op/s 1.954ms 6.067ms 1779.281op/s
scenario:load:petclinic:iast:high_load worse
[+454.953µs; +1308.686µs] or [+2.614%; +7.520%]		 worse
[+0.575ms; +2.010ms] or [+2.030%; +7.102%]		 unstable
[-43.452op/s; +18.327op/s] or [-16.537%; +6.975%]		 18.284ms 29.600ms 250.188op/s 17.402ms 28.308ms 262.750op/s
scenario:load:petclinic:code_origins:high_load better
[-1.486ms; -0.595ms] or [-8.079%; -3.235%]		 better
[-2.022ms; -0.601ms] or [-6.822%; -2.028%]		 unstable
[-16.834op/s; +45.147op/s] or [-6.762%; +18.133%]		 17.350ms 28.333ms 263.125op/s 18.390ms 29.645ms 248.969op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~64af64756a, baseline=1.61.0-SNAPSHOT~b664cc0f2a
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.23 ms) : 1218, 1241
.   : milestone, 1230,
iast (3.246 ms) : 3199, 3293
.   : milestone, 3246,
iast_FULL (6.143 ms) : 6080, 6206
.   : milestone, 6143,
iast_GLOBAL (3.726 ms) : 3660, 3791
.   : milestone, 3726,
profiling (2.555 ms) : 2529, 2581
.   : milestone, 2555,
tracing (1.927 ms) : 1910, 1945
.   : milestone, 1927,
section candidate
no_agent (1.25 ms) : 1237, 1262
.   : milestone, 1250,
iast (3.332 ms) : 3285, 3379
.   : milestone, 3332,
iast_FULL (6.038 ms) : 5977, 6100
.   : milestone, 6038,
iast_GLOBAL (3.626 ms) : 3568, 3685
.   : milestone, 3626,
profiling (2.268 ms) : 2248, 2289
.   : milestone, 2268,
tracing (1.85 ms) : 1834, 1865
.   : milestone, 1850,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.23 ms [1.218 ms, 1.241 ms] -
iast 3.246 ms [3.199 ms, 3.293 ms] 2.016 ms (164.0%)
iast_FULL 6.143 ms [6.08 ms, 6.206 ms] 4.913 ms (399.6%)
iast_GLOBAL 3.726 ms [3.66 ms, 3.791 ms] 2.496 ms (203.0%)
profiling 2.555 ms [2.529 ms, 2.581 ms] 1.325 ms (107.8%)
tracing 1.927 ms [1.91 ms, 1.945 ms] 697.9 µs (56.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.25 ms [1.237 ms, 1.262 ms] -
iast 3.332 ms [3.285 ms, 3.379 ms] 2.082 ms (166.6%)
iast_FULL 6.038 ms [5.977 ms, 6.1 ms] 4.788 ms (383.1%)
iast_GLOBAL 3.626 ms [3.568 ms, 3.685 ms] 2.377 ms (190.2%)
profiling 2.268 ms [2.248 ms, 2.289 ms] 1.018 ms (81.5%)
tracing 1.85 ms [1.834 ms, 1.865 ms] 599.735 µs (48.0%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~64af64756a, baseline=1.61.0-SNAPSHOT~b664cc0f2a
    dateFormat X
    axisFormat %s
section baseline
no_agent (19.44 ms) : 19241, 19638
.   : milestone, 19440,
appsec (18.536 ms) : 18349, 18723
.   : milestone, 18536,
code_origins (18.744 ms) : 18558, 18931
.   : milestone, 18744,
iast (17.754 ms) : 17577, 17930
.   : milestone, 17754,
profiling (18.605 ms) : 18421, 18789
.   : milestone, 18605,
tracing (18.002 ms) : 17824, 18179
.   : milestone, 18002,
section candidate
no_agent (19.384 ms) : 19187, 19580
.   : milestone, 19384,
appsec (18.566 ms) : 18375, 18756
.   : milestone, 18566,
code_origins (17.734 ms) : 17560, 17909
.   : milestone, 17734,
iast (18.653 ms) : 18466, 18839
.   : milestone, 18653,
profiling (18.697 ms) : 18511, 18884
.   : milestone, 18697,
tracing (17.778 ms) : 17600, 17955
.   : milestone, 17778,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 19.44 ms [19.241 ms, 19.638 ms] -
appsec 18.536 ms [18.349 ms, 18.723 ms] -903.353 µs (-4.6%)
code_origins 18.744 ms [18.558 ms, 18.931 ms] -695.099 µs (-3.6%)
iast 17.754 ms [17.577 ms, 17.93 ms] -1.686 ms (-8.7%)
profiling 18.605 ms [18.421 ms, 18.789 ms] -834.354 µs (-4.3%)
tracing 18.002 ms [17.824 ms, 18.179 ms] -1.438 ms (-7.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 19.384 ms [19.187 ms, 19.58 ms] -
appsec 18.566 ms [18.375 ms, 18.756 ms] -818.3 µs (-4.2%)
code_origins 17.734 ms [17.56 ms, 17.909 ms] -1.65 ms (-8.5%)
iast 18.653 ms [18.466 ms, 18.839 ms] -731.369 µs (-3.8%)
profiling 18.697 ms [18.511 ms, 18.884 ms] -686.788 µs (-3.5%)
tracing 17.778 ms [17.6 ms, 17.955 ms] -1.606 ms (-8.3%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mhlidd/run_spock_with_strict
git_commit_date 1775497568 1775503219
git_commit_sha b664cc0 64af647
release_version 1.61.0-SNAPSHOT~b664cc0f2a 1.61.0-SNAPSHOT~64af64756a
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1775505346 1775505346
ci_job_id 1570881601 1570881601
ci_pipeline_id 106256212 106256212
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-p8ynegt3 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-p8ynegt3 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~64af64756a, baseline=1.61.0-SNAPSHOT~b664cc0f2a
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.025 s) : 15025000, 15025000
.   : milestone, 15025000,
appsec (14.732 s) : 14732000, 14732000
.   : milestone, 14732000,
iast (18.687 s) : 18687000, 18687000
.   : milestone, 18687000,
iast_GLOBAL (18.101 s) : 18101000, 18101000
.   : milestone, 18101000,
profiling (15.068 s) : 15068000, 15068000
.   : milestone, 15068000,
tracing (14.772 s) : 14772000, 14772000
.   : milestone, 14772000,
section candidate
no_agent (15.394 s) : 15394000, 15394000
.   : milestone, 15394000,
appsec (14.795 s) : 14795000, 14795000
.   : milestone, 14795000,
iast (18.162 s) : 18162000, 18162000
.   : milestone, 18162000,
iast_GLOBAL (17.82 s) : 17820000, 17820000
.   : milestone, 17820000,
profiling (15.319 s) : 15319000, 15319000
.   : milestone, 15319000,
tracing (14.693 s) : 14693000, 14693000
.   : milestone, 14693000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.025 s [15.025 s, 15.025 s] -
appsec 14.732 s [14.732 s, 14.732 s] -293.0 ms (-2.0%)
iast 18.687 s [18.687 s, 18.687 s] 3.662 s (24.4%)
iast_GLOBAL 18.101 s [18.101 s, 18.101 s] 3.076 s (20.5%)
profiling 15.068 s [15.068 s, 15.068 s] 43.0 ms (0.3%)
tracing 14.772 s [14.772 s, 14.772 s] -253.0 ms (-1.7%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.394 s [15.394 s, 15.394 s] -
appsec 14.795 s [14.795 s, 14.795 s] -599.0 ms (-3.9%)
iast 18.162 s [18.162 s, 18.162 s] 2.768 s (18.0%)
iast_GLOBAL 17.82 s [17.82 s, 17.82 s] 2.426 s (15.8%)
profiling 15.319 s [15.319 s, 15.319 s] -75.0 ms (-0.5%)
tracing 14.693 s [14.693 s, 14.693 s] -701.0 ms (-4.6%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~64af64756a, baseline=1.61.0-SNAPSHOT~b664cc0f2a
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.487 ms) : 1476, 1499
.   : milestone, 1487,
appsec (3.744 ms) : 3527, 3962
.   : milestone, 3744,
iast (2.269 ms) : 2200, 2338
.   : milestone, 2269,
iast_GLOBAL (2.313 ms) : 2244, 2382
.   : milestone, 2313,
profiling (2.528 ms) : 2356, 2701
.   : milestone, 2528,
tracing (2.082 ms) : 2028, 2135
.   : milestone, 2082,
section candidate
no_agent (1.498 ms) : 1486, 1509
.   : milestone, 1498,
appsec (3.841 ms) : 3618, 4064
.   : milestone, 3841,
iast (2.278 ms) : 2209, 2347
.   : milestone, 2278,
iast_GLOBAL (2.314 ms) : 2245, 2383
.   : milestone, 2314,
profiling (2.121 ms) : 2065, 2178
.   : milestone, 2121,
tracing (2.07 ms) : 2017, 2123
.   : milestone, 2070,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.487 ms [1.476 ms, 1.499 ms] -
appsec 3.744 ms [3.527 ms, 3.962 ms] 2.257 ms (151.8%)
iast 2.269 ms [2.2 ms, 2.338 ms] 781.58 µs (52.6%)
iast_GLOBAL 2.313 ms [2.244 ms, 2.382 ms] 825.854 µs (55.5%)
profiling 2.528 ms [2.356 ms, 2.701 ms] 1.041 ms (70.0%)
tracing 2.082 ms [2.028 ms, 2.135 ms] 594.442 µs (40.0%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.498 ms [1.486 ms, 1.509 ms] -
appsec 3.841 ms [3.618 ms, 4.064 ms] 2.344 ms (156.5%)
iast 2.278 ms [2.209 ms, 2.347 ms] 780.776 µs (52.1%)
iast_GLOBAL 2.314 ms [2.245 ms, 2.383 ms] 816.225 µs (54.5%)
profiling 2.121 ms [2.065 ms, 2.178 ms] 623.839 µs (41.7%)
tracing 2.07 ms [2.017 ms, 2.123 ms] 572.405 µs (38.2%)

@mhlidd mhlidd added type: enhancement Enhancements and improvements comp: testing Testing comp: config Configuration labels Apr 6, 2026
@mhlidd mhlidd force-pushed the mhlidd/run_spock_with_strict branch from 5ba6f69 to b0147c9 Compare April 6, 2026 17:00
@mhlidd mhlidd force-pushed the mhlidd/run_spock_with_strict branch from 95b7e3f to 82a89e0 Compare April 6, 2026 18:20
@mhlidd mhlidd marked this pull request as ready for review April 6, 2026 20:51
@mhlidd mhlidd requested review from a team as code owners April 6, 2026 20:51
@vandonr
Copy link
Copy Markdown
Contributor

vandonr commented Apr 7, 2026

@codex review

@vandonr
Copy link
Copy Markdown
Contributor

vandonr commented Apr 7, 2026

ok so most of the changes are coming from the test files where we override that global config

@chatgpt-codex-connector
Copy link
Copy Markdown

chatgpt-codex-connector bot commented Apr 7, 2026

Codex Review: Didn't find any major issues. What shall we delve into next?

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mhlidd mhlidd added this pull request to the merge queue Apr 7, 2026
@dd-octo-sts
Copy link
Copy Markdown
Contributor

dd-octo-sts bot commented Apr 7, 2026

/merge

@gh-worker-devflow-routing-ef8351
Copy link
Copy Markdown

gh-worker-devflow-routing-ef8351 bot commented Apr 7, 2026 •
edited
Loading

View all feedbacks in Devflow UI.

2026-04-07 15:06:47 UTC ℹ️ Start processing command /merge
Use /merge -c to cancel this operation!

2026-04-07 15:06:52 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).

Use /merge -c to cancel this operation!

2026-04-07 16:11:56 UTC ℹ️ MergeQueue: Readding this merge request to the queue because another merge request processed with yours failed. No action is needed from your side.
Use /merge -c to cancel this operation!

⏳ Processing

@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vandonr vandonr vandonr approved these changes

@manuel-alvarez-alvarez manuel-alvarez-alvarez Awaiting requested review from manuel-alvarez-alvarez manuel-alvarez-alvarez is a code owner automatically assigned from DataDog/asm-java

@claponcet claponcet Awaiting requested review from claponcet claponcet is a code owner automatically assigned from DataDog/asm-java

Assignees

No one assigned

Labels

comp: config Configuration comp: testing Testing type: enhancement Enhancements and improvements

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mhlidd @vandonr